COMPUTER  SCIENCE  &  TECHNOLOGY: 


TECHNOLOGY  ASSESSMENT: 
ADP  INSTALLATION  PERFORMANCE 
MEASUREMENT  AND  REPORTING 


NBS  Special  Publication  500-53 

U.S.  DEPARTMENT  OF  COMMERCE 
National  Bureau  of  Standards 


NATIONAL  BUREAU  OF  STANDARDS 


The  National  Bureau  of  Standards'  was  established  by  an  act  ot  Congress  on  March  3,  1901. 
The  Bureau's  overall  goal  is  to  strengthen  and  advance  the  Nation's  science  and  technology 
and  facilitate  their  effective  application  for  public  benefit.  To  this  end,  the  Bureau  conducts 
research  and  provides:  (1)  a  basis  for  the  Nation's  physical  measurement  system,  (2)  scientific 
and  technological  services  for  industry  and  government,  (3)  a  technical  basis  for  equity  in 
trade,  and  (4)  technical  services  to  promote  public  safety.  The  Bureau's  technical  work  is  per- 
formed by  the  National  Measurement  Laboratory,  the  National  Engineering  Laboratory,  and 
the  Institute  for  Computer  Sciences  and  Technology. 

THE  NATIONAL  MEASUREMENT  LABORATORY  provides  the  national  system  of 
physical  and  chemical  and  materials  measurement;  coordinates  the  system  with  measurement 
systems  of  other  nations  and  furnishes  essential  services  leading  to  accurate  and  uniform 
physical  and  chemical  measurement  throughout  the  Nation's  scientific  community,  industry, 
and  commerce;  conducts  materials  research  leading  to  improved  methods  of  measurement, 
standards,  and  data  on  the  properties  of  materials  needed  by  industry,  commerce,  educational 
institutions,  and  Government;  provides  advisory  and  research  services  to  other  Government 
agencies;  develops,  produces,  and  distributes  Standard  Reference  Materials;  and  provides 
calibration  services.  The  Laboratory  consists  of  the  following  centers: 

Absolute  Physical  Quantities^  —  Radiation  Research  —  Thermodynamics  and 
Molecular  Science  —  Analytical  Chemistry  —  Materials  Science. 

THE  NATIONAL  ENGINEERING  LABORATORY  provides  technology  and  technical  ser- 
vices to  the  public  and  private  sectors  to  address  national  needs  and  to  solve  national 
problems;  conducts  research  in  engineering  and  applied  science  in  support  of  these  efforts; 
builds  and  maintains  competence  in  the  necessary  disciplines  required  to  carry  out  this 
research  and  technical  service;  develops  engineering  data  and  measurement  capabilities; 
provides  engineering  measurement  traceability  services;  develops  test  methods  and  proposes 
engineering  standards  and  code  changes;  develops  and  proposes  new  engineering  practices; 
and  develops  and  improves  mechanisms  to  transfer  results  of  its  research  to  the  ultimate  user. 
The  Laboratory  consists  of  the  following  centers: 

Applied  Mathematics  —  Electronics  and  Electrical  Engineering^  —  Mechanical 
Engineering  and  Process  Technology^  —  Building  Technology  —  Fire  Research  — 
Consumer  Product  Technology  —  Field  Methods. 

THE  INSTITUTE  FOR  COMPUTER  SCIENCES  AND  TECHNOLOGY  conducts 
research  and  provides  scientific  and  technical  services  to  aid  Federal  agencies  in  the  selection, 
acquisition,  application,  and  use  of  computer  technology  to  improve  effectiveness  and 
economy  in  Government  operations  in  accordance  with  Public  Law  89-306  (40  U.S.C.  759), 
relevant  Executive  Orders,  and  other  directives;  carries  out  this  mission  by  managing  the 
Federal  Information  Processing  Standards  Program,  developing  Federal  ADP  standards 
guidelines,  and  managing  Federal  participation  in  ADP  voluntary  standardization  activities; 
provides  scientific  and  technological  advisory  services  and  assistance  to  Federal  agencies;  and 
provides  the  technical  foundation  for  computer-related  policies  of  the  Federal  Government. 
The  Institute  consists  of  the  following  centers: 

Programming  Science  and  Technology  —  Computer  Systems  Engineering. 

'Headquarters  and  Laboratories  at  Gaithersburg,  MD,  unless  otherwise  noted; 
mailing  address  Washington,  DC  20234. 

'Some  divisions  within  the  center  are  located  at  Boulder,  CO  80303. 


COMPUTER  SCIENCE  &  TECHNOLOGY: 

TECHNOLOGY  ASSESSMENT: 

ADP  INSTALLATION  PERFORMANCE 

MEASUREMENT  AND  REPORTING 


U.S.  DEPARTMENT  OF  COMMERCE,  Juanita  M.  Kreps,  Secretary 
Luther  H.  Hodges,  Jr.,  Under  Secretary 

Jordan  J.  Baruch,  Assistant  Secretary  for  Science  and  Technology 
NATIONAL  BUREAU  OF  STANDARDS.  Ernest  Ambler.  Director 

Issued  September  1979 


Carol  B.  Wilson 


Institute  for  Computer  Sciences  and  Technology 
National  Bureau  of  Standards 
Washington,  DC  20234 


Reports  on  Computer  Science  and  Technology 

The  National  Bureau  of  Standards  has  a  special  responsibihty  within  the  Federal 
Government  for  computer  science  and  technology  activities.  The  programs  of  the 
NBS  Institute  for  Computer  Sciences  and  Technology  are  designed  to  provide  ADP 
standards,  guidelines,  and  technical  advisory  services  to  improve  the  effectiveness  of 
computer  utilization  in  the  Federal  sector,  and  to  perform  appropriate  research  and 
development  efforts  as  foundation  for  such  activities  and  programs.  This  publication 
series  will  report  these  NBS  efforts  to  the  Federal  computer  community  as  well  as  to 
interested  specialists  in  the  academic  and  private  sectors.  Those  wishing  to  receive 
notices  of  publications  in  this  series  should  complete  and  return  the  form  at  the  end 
of  this  publication. 


National  Bureau  of  Standards  Special  Publication  500-53 

Nat.  Bar.  Stend.  (U.S.)  Spec.  Publ.  500-53,  37  pages  (Sept.  1979) 
CODEN:  XNBSAV 


Library  of  Congress  Catalog  Card  Number:  79-600154 


U.S.  GOVERNMENT  PRINTING  OFFICE 
WASHINGTON:  1^79 


For  sale  by  the  Superintendent  of  Documents,  U.S.  Government  Printing  Office,  Washington,  D.C.  20402 
Stock  No.  003-003-02123-7    Price  $2.00 
(Add  25  pereent  adiiltioinl  for  other  than  U.S.  mailing) 


Technology  Assessment: 
ADP  Installation  Performance 
Measurement  and  Reporting 


Table  of  Contents 

1.0    Introduction    1 

1.1  Overview   1 

1.2  ADP  Installation  Model    2 

2.0    Installation  Performance  Management     ...  4 

2.1  Upper  Management     ..    4 

2.2  User  Community     .   5 

2.2.1  Measuring  User  Satisfaction     ...  5 

2.2.2  Setting  Service  Levels    5 

2.2.3  Measuring  Service    6 

2.2.4  Reporting  to  Users  ........  6 

2.2.5  Correcting  Poor  Service    7 

2.3  Resource  Management    8 

2.3.1  Operational  Standards    8 

2.3.2  Resource  Utilization  Standards  .   .  9 

2.3.3  Forecasting    9 

2.4  Vendor   10 

2.5  Summary   11 

3.0    Installation  Performance  Activities  in  the 

Federal  Sector    12 

3.1  Auditor  Perspective    12 

3.2  Installation  Perspective    14 

3.3  Summary   15 

4.0    Current  Inadequacies  in  Installation 

Performance  Practices    16 

4.1  Lack  of  Measurabil  i ty   16 

4.2  Lack  of  Well-Defined  Terms   17 

4.3  Lack  of  "Good"  Values   17 

4.4  Lack  of  Statistical  CPE  Techniques     .   .  18 

5.0    Proposed  Remedial  Work  .   .   .   .   ,   20 

5.1  Standard  Performance  Measures    20 

5.2  Data  Base  for  Normative  Performance  .   .  20 

5.3  Development  of  Statistical  CPE 
Techniques   21 

5.4  Summary   21 

6.0    Bibliography   22 

Appendix  A  -  Informational  Model    27 


iii 


ACKNOWLEDGMENTS 


The  author  wishes  to  thank  Peter  J.  Calomeris  and  Wayne  D. 
Bennett  of  the  National  Bureau  of  Standards  for  their 
contributions  to  this  report.  Mr.  Calomeris  performed  the 
literature  search,  provided  copies  of  all  the  references 
used  in  this  report,  and  prepared  the  bibliography.  Mr. 
Bennett  performed  the  auditing  agency  site  visits  and 
contributed  to  the  material  in  Appendix  A. 


i  V 


Technology  Aissessment: 
ADP  Installation  Performance 
Measurement  and  Reporting 


Carol  B.  Wilson 


Abstract 


This  report  compares  the  current  status  of  ADP 
installation  performance  measurement  and  reporting  in 
the  Federal  ADP  community  to  the  best  practices  as 
found  in  the  Federal  and  private  sectors  and  described 
in  the  literature.     The  comparison  reveals  that  more 
effort  could  be  expended  by  the  Federal  sites  in  the 
area  of  computer  performance  management.     The  principal 
obstacles  to  more  and  better  performance  programs  are 
perceived  to  be  the  lack  of  needed  measures  on  many 
systems  and  the  magnitude  of  the  effort  involved  in 
accessing  and  analyzing  the  measures  which  are  avail- 
able.    The  report  discusses  several  underlying  causes 
for  these  obstacles  and  makes  three  recommendations 
which  could  partially  relieve  the  situation:     (1)  de- 
velopment of  standard  performance  measures,   (2)  devel- 
opnent  of  a  Government-wide  data  base  for  normative 
performance  ranges,  and   (3)  development  of  statistical 
computer  performance  evaluation  techniques. 

Key  words:     Computer  performance  evaluation  (CPE); 
computer  performance  management  (CPM) ;  installation 
management;     installation  performance  management; 
performance  measurement  and  reporting;  resource 
management;     standard  performance  measures. 


1.0  Introduction 
1.1  Overview 

The  purpose  of  this  report  is  to 
examine  the  current  status  of  ADP 
installation  performance  measurement 
and  reporting  in  the  Federal  ADP 
community,  to  identify  and  discuss  any 
current  inadequacies  in  practice,  and 
to  make  recommendations  for  the  cor- 
rection of  those  inadequacies.  For 
purposes  of  this  report,  an  ADP 
installation  is  considered  to  be  the 
complete  set  of  resources  dedicated  to 
meeting  the  automated  processing  needs 


of  an  agency.     Typically,  the  ADP 
installation  consists  of  the  computer 
hardware  itself  along  with  the 
administrative,  systems  support,  user 
support,  and  operational  staff. 

The  major  purpose  of  an  ADP 
installation  should  be  to  meet  the 
computing  needs  of  its  user  population 
at  the  least  cost.    While  this  qoal 
may  be  simply  stated,  the  measurement 
of  the  degree  of  success  in  meeting 
this  goal  is  a  complex  problem. 
Ideally  we  would  like  to  be  able  to 
solve  this  problem  by  using  precise 
quantitative  measurements  that  are 
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objective  and  free  of  human  biases  and 
intuitive  judgments.     This  is  not 
possible  because  the  performance  we 
are  trying  to  measure  depends  on 
complex  human  behavior  and  also  on  the 
complexities  of  modern  computer  system 
relationships  which  are  not  fully 
understood.     In  addition,  randomly 
varying  workloads,  or  even  cyclical 
workloads,  pose  problems  in  optimizing 
service  and  cost. 

Further,  most  of  today's  ADP 
installations  must  meet  the  demands  of 
a  multi-user,  multi-application 
environment.     The  work  handled  by  the 
installation  is  a  composite  of  the 
computing  needs  of  all  of  the  usersy 
each  putting  different  demands  on  the 
installation.     Because  the  demand  for 
resources  varies  by  user,  improving 
the  performance  of  some  resources  for 
the  benefit  of  specific  users  may  be 
at  the  expense  of  other  users  or  may 
even  have  an  overall  detrimental 
effect  on  installation  performance. 
Hence,  the  goal  of  an  installation 
cannot  be  to  provide  optimal  service 
to  each  user  but  to  provide  "good" 
service  to  all  users  with  respect  to 
organizational  goals  and  priorities. 
It  should  be  noted  that  state-of-the- 
art  techniques,  n-th  generation 
machines  and  higher  level  languages  do 
not  necessarily  result  in  a  "good" 
installation  but  are  only  important  in 
that  they  enhance  the  potential  for 
better  productivity,  lower  cost,  or 
better  user  service. 


To  meet  the  goal  of  providing 
good  service  at  the  least  cost,  the 
installation  manager  must  do  capacity 
planning.     Gilmore  [GILMM  78-pl] 
defines  capacity  as  "the  workload  that 
a  well- tuned  system  can  process  with- 
out exceeding  the  limits  of  [defined 
by]   user  performance  objectives." 
Capacity  planning  can  thus  be  thought 
of  as  optimizing  the  installation 
resources  (e.g.,  tuning  the  system)  to 
the  user  workload  at  the  least  cost, 
for  both  the  short  and  long  terms.  To 
be  successful  in  this  function,  the 
ADP  manager  must  undertake  several 
tasks:     (1)  communicate  with  upper 
management,   (2)   serve  and  influence 
the  user  community,   (3)  manage  his 
resources,  and   (4)  maintain  vendor 
relations.     A  Computer  Performance 
Management  (CPM)   program  supports  the 
ADP  manager  in  all  of  these  functions 
by  providing  quantitative  data  for  the 
decision-making  process.    We  will 
discuss  these  tasks  and  how  they 
relate  to  installation  performance  by 
introducing  a  model  below. 


1.2  ADP  Installation  Model 

The  model  below  represents  a 
functional  view  of  an  ADP  installation 
as  seen  from  the  perspective  of  the 
ADP  installation  manager. 


UPPER  MANAGEMENT 

ADP  INSTALLATION 
USER   —   MANAGER  —  ' —  VENDOR 

RESOURCES 
Figure  1:  ADP  Installation  Model 
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We  choose  to  place  the  instal- 
lation manager  in  the  center  of  the 
model  because  he/she  controls  or 
influences  the  factors  which  contri- 
bute to  the  success  or  failure  of  the 
ADP  installation.     Although  the  prob- 
lems and  responsibilities  that  the  ADP 
manager  must  face  are  similar  to  those 
encountered  by  managers  in  other  func- 
tional areas  in  an  organization,   it  is 
becoming  more  widely  recognized  that 
there  are  extra  dimensions  to  the 
control  of  the  ADP  installation.  The 
ADP  manager  must  be  accountable  to 
upper  management  for  the  economic 
operation  of  the  installation.     He  is 
responsible  to  the  users  for  providing, 
an  efficient  and  effective  service. 
He  must  oversee  the  actions  and 
obligations  of  the  vendor.  And 
finally,  he  must  control  the  resources 
of  his  installation.     However,  the 
on-rushing  technological  changes  and 
complexities  of  the  computer  field 
affect  the  ADP  manager's  ability  to 
meet  these  responsibilities.  For 
example,  there  is  rarely  a  large  set 
of  precedents  to  aid  him  in  making 
many  of  the  short  or  long  range 
•decisions  which  often  arise.  Second- 
ly, staffing  is  uncertain  as  new 
technologies  require  new  skills. 
Further,  the  ADP  manager  is  required 
;to  assume  multiple  roles,  to  manage  a 
multi-million  dollar  facility  upon 
which  the  success  of  many  agency 
programs  depends  and  to  remain 
technically  knowledgeable  in  a  rapidly 
changing  technology.     And  finally,  he 
must  also  oversee  the  mundane 
activities  which  can  so  easily  degrade 
installation  performance  if  left 
unattended . 

In  Section  2  we  briefly  discuss 
each  of  the  four  areas  of  our  model  by 
describing  the  functions  an  ADP  mana- 
•ger  must  undertake  to  fulfill  his 
obligations  with  respect  to  each  area: 
upper  management,  users,  resources, 
and  vendors.     In  Section  3  we  discuss 
the  current  practices  in  the  Federal 
sector.     The  discussion  notes  that  few 
ADP  installations  fully  address  the 
four  areas  of  concern.     In  particular, 
the  upper  management  relationship  is 
quite  weak.     Planning  and  managing  of 
installation  resources  has  typically 
digressed  to  a  fire-fighting  mode  of 
operation  at  many  sites. 


Section  4  discusses  possible 
underlying  causes  for  this  lack  of 
good  resource  management.  These 
causes  include  current  inadequacies 
with  respect  to  the  measur abil i ty  of 
computer  systems,  the  lack  of  well- 
defined  terms  for  performance  and 
utilization  measures,  the  lack  of 
"good"  values  for  installation  per- 
formance, and  the  lack  of  statistical 
techniques  which  have  been  validated 
for  use  in  the  computer  environment 
and  which  can  be  easily  used  by  non- 
statisticians.     The  upper  management 
problem  is  not  addressed  since  its 
solution  lies  in  the  area  of  manage- 
ment science,  not  computer  science. 
Section  5  recommends  work  which  could 
be  performed  to  increase  installation 
performance  measurement  and  reporting: 
development  of  standard  performance 
measures,  generation  of  a  data  base 
for  normative  performance,  and  the 
development  of  "cookbooks"  for 
statistical  techniques  to  be  used  in 
the  evaluation  of  computer  systems. 
The  thrust  of  the  three  recommen- 
dations which  are  made  is  that  more 
effort  would  be  expended  by  Federal 
sites  in  the  area  of  CPM  if  the  task 
did  not  require  so  much  personnel  time 
and  expertise. 
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2.0  Installation  Performance 
Management 

Before  we  address  the  current 
state  of  installation  performance 
practices  in  the  Federal  sector,  we 
need  to  establish  a  basis  for  discus- 
sion.    In  this  section  we  will  de- 
scribe the  best  practices  as  found  in 
the  Federal  and  private  sectors  and 
described  in  the  literature.  Section 
6,0  (Bibliography)   contains  a  list  of 
literature  references  which  were 
reviewed  as  part  of  this  study.  Each 
of  the  four  relationships  of  concern 
—  upper  management,  user  community, 
resources,  and  vendor        and  the 
corresponding  functions  will  be 
discussed.     These  functions  should  be 
performed  by  the  ADP  manager  to  ensure 
that  his  installation  is  meeting  its 
goal  of  providing  good  service  to  its 
users  at  the  least  cost.     Then,  having 
established  this  baseline  for  instal- 
lation performance  practices,  we  will 
be  able  (in  Section  3)   to  compare 
current  practices  in  the  Federal 
sector  with  the  baseline. 

We  will  restrict  our  description 
to  the  more  traditional  ADP  instal- 
lation       centralized  batch  with 
timesharing  and  remote  batch.  The 
question  of  informational  needs  for 
networks  and  other  forms  of  distri- 
buted processing  is  beyond  the  scope 
of  this  report.     However,  within  the 
best  practices  as  set  forth  below,  we 
will  cJiscuss  functions  which  could  be 
performed  by  the  manager  of  any  size 
ADP  installation  whether  it  is 
oriented  toward  data  processing  or 
toward  research  and  development.  Of 
course  the  size  of  the  installation, 
its  position  in  the  organization,  and 
its  mission  will  cause  variations  in 
the  details  in  carrying  out  the 
functions  given  below,  although  most 
of  the  tasks  are  necessary  for  any  ADP 
installation  to  be  well-managed  and 
operated , 


2.1  Upper  Management 

The  ADP  installation  manager  has 
a  responsibility  to  report  to  his 
upper  management  on  the  status  of  re- 
quirements, performance,  and  finances 
of  the  ADP  installation.     It  is  best 
that  the  installation  manager  report 


to  a  person  who  has  sufficient 
authority  to  ensure  that  the  ADP 
installation  will  receive  adequate 
support  for  its  policies.     It  is 
necessary  that  the  installation  man- 
ager receive  this  support  from  upper 
management  for  the  controls,  proce- 
dures, and  management  policies  which 
he  establishes  at  the  installation  for 
the  assignment  of  resources  to  the 
users.     To  help  him  in  this  task,  he 
should  present  performance  information 
to  upper  management  that  is  at  a  level 
of  abstraction  which  can  be  easily 
digested  and  understood,  but  is  suf- 
ficient to  support  the  upper  manage- 
ment decision-making  process. 

To  permit  the  ADP  installation  to 
fully  support  the  goals  and  objectives 
of  the  organization,  upper  management 
must  make  the  effort  to  include  the 
ADP  manager  in  the  planning  process, 
to  recognize  the  impact  that  organi- 
zational plans  will  have  on  the  ADP 
installation,  and  to  know  where  the 
ADP  installation  fits  into  the  plans. 
To  aid   in  the  planning  process,  upper 
management  needs  to  know  the  costs  of 
various  levels  of  service  and  their 
effects  on  program  schedules,  person- 
nel levels,  and  budget.     The  ADP 
manager  must,  therefore,  relate  the 
cost  of  providing  service  at  various 
levels  to  the  current  configuration 
and,  perhaps,  to  future  configur- 
ations.    The  ADP  manager  should  pro- 
ject the  load  which  planned  projects 
will  place  on  installation  resources. 
The  ADP  manager  should  encourage 
management  to  reconsider  committing  to 
ill-conceived  or  unrealistic  ADP 
projects.     If  the  ADP  manager  is 
successful  in  this  task,  he  can  save 
his  installation  from  much  of  the  busy 
work  which  has  minimal  payoff  but 
which  is  a  constant  drain  on 
resources . 

The  ADP  manager  should  keep  his 
upper  management  informed  about  the 
current  operations  and  the  performance 
of  the  ADP  installation  with  respect 
to  organizational  goals  and  object- 
ives.    That  is  to  say,  reports  should 
be  concerned  with  the  effectiveness  of 
the  installation;     i.e.,  how  well  the 
installation  is  satisfying  the  user 
requirements  for  service.  Depending 
upon  the  management  style  of  the 
organization,  upper  management  may  be 
interested  in  regular  reports  or  may 
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only  be  interested  in  the  reporting  of 
exceptions  to  the  expected  service 
levels.     The  exact  content  of  the 
reports  would  also  vary  with  manage- 
ment style  but  could  address  the 
amount  of  work  being  done  by  the 
various  departments  and  the  instal- 
lation's ability  to  process  the  work 
in  a  timely  manner.     Reports  should 
also  be  presented  which  are  concerned 
with  budget,  revenue,  actual  cost,  and 
financial  tracking  for  the  instal- 
lation.    Reports  concerning  component 
utilization,  etc.     are  probably 
inappropriate  for  upper  management, 
although  certainly  necessary  for 
proper  resource  management. 


2.2  User  Community 

The  relationship  between  the  ADP 
manager  and  the  user  community  is  of 
crucial  importance  to  the  success  of 
the  installation.     The  ADP  instal- 
lation exists  to  serve  its  users,  but 
it  must  also  influence  their  behavior 
if  the  installation  is  to  be  both 
effective  and  efficient.     The  ADP 
manager's  most  difficult  task  probably 
lies  in  working  with  and  for  the  user 
community.     The  expertise  in  a  typical 
user  community  often  ranges  from  the 
very  naive  to  the  very  sophisticated 
user.     But  even  the  most  sophisticated 
users  usually  have  only  the  vaguest 
concept  of  the  overall  installation 
operation  and  the  functions  which  the 
ADP  manager  performs.     The  ADP  manager 
must  have  rapport  with  his  users  and 
instill  in  them  a  sense  of  mutual 
cooperation  and  good  will. 

There  are  several  general 
measures  of  installation  performance 
which  involve  the  user  community. 
They  are  based  upon  the  acceptance  of 
the  computer  as  reflected  in  the 
number  of  departments  using  the 
facility,  the  use  each  department 
makes  of  it,  and  how  well  the  computer 
is  accepted  as  an  easy-to-use  tool. 
But  these  measures  only  provide  a 
rough  estimation  of  user  acceptance 
and  do  not  indicate  user  satisfaction 
with  the  installation.     For  example, 
many  departments  may  use  the  facility 
with  ease  but  still  not  be  satisfied 
with  the  installation  because  service 
is  poor.     User  satisfaction  can  be  one 
of  the  best  general  indicators  of 


installation  performance.     An  itera- 
tive plan  for  ensuring  user  satisfac- 
tion is  detailed  below.     It  calls  for 
measuring  the  current  level  of  user 
satisfaction,  setting  service  levels, 
measuring  the  service,  reporting  to 
the  users  about  the  success  or  failure 
in  meeting  the  service  level  goals, 
and  taking  corrective  action  for 
missed  goals. 

2.2.1  Measuring  User  Satisfaction 

One  of  the  accepted  means  of 
ascertaining  user  satisfaction  is  to 
use  a  well-designed  survey.  Develop- 
ing a  good  survey  may  be  hampered  by 
two  facts:     (1)   satisfaction  is 
related  to  the  user's  expectation, 
which  may  rise  or  fall  rapidly  with 
the  occurrence  of  a  single  event,  and 
(2)   satisfaction  is  not  a  single 
quantity  but  a  blend  of  many  factors. 
Hence  the  survey  must  allow  many 
factors  to  be  evaluated  simultaneously 
and  provide  a  means  of  ensuring  reli- 
ability of  the  data,  but  must  not  be 
excessively  long  lest  errors  be  intro- 
duced through  fatigue.     Pearson  [PEARS 
77]   discusses  the  requirements  for  a 
survey  designed  to  measure  user  satis- 
faction and  the  procedures  he  used  to 
develop  a  comprehensive  thirty-nine" 
factor  questionnaire  which  takes  only 
thirty  minutes  to  complete.     He  dis- 
cusses the  factors  used   in  his  ques- 
tionnaire and  why  they  are  important 
to  computer  users. 

2.2.2  Setting  Service  Levels. 

The  typical  user  is  principally 
concerned  about  the  service  he  re- 
ceives in  meeting  his  project  dead- 
lines and  the  consistency  of  the 
service.     If  he  is  regularly  receiving 
poor  service  he  is  little  consoled  by 
the  fact  that  everyone  else  is  more 
fortunate  than  he.     It  is  unusual  for 
one  user  to  receive  consistently 
poorer  service  than  other  users,  but 
it  is  not  unusual  for  him  to  remember 
only  the  reports  that  were  late,  the 
jobs  that  did  not  come  back  when  he 
wanted  them,  or  the  day  that  time- 
sharing response  was  abysmal.     With  no 
defined  levels  of  service  there  is  no 
basis  for  the  ADP  manager  or  the  users 
to  judge  the  installation.     Hence,  in 
.order  to  satisfy  the  users,   it  is 
essential  that  the  ADP  manager  know 
what  the  users'   service  needs  really 
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are.     The  ADP  manager  and  the  user 
community  must  together  establish 
service  levels  for  timeliness,  avail- 
ability, accuracy  and  cost  which  are 
acceptable  to  the  users  (or  classes  of 
users)   and  which  are  reasonable  to  the 
ADP  manager.     These  service  levels 
then  provide  one  basis  for  measuring 
installation  performance.  Depending 
on  the  type  of  installation  and  users, 
the  service  levels  may  be  categorized 
by  class  of  job:     batch,  timesharing, 
etc.;     by  class  of  user:  payroll, 
accounts,  etc.?     by  resource  class: 
requested  and  used;     or  by  some  other 
category.     The  categories  should 
reflect  the  needs  of  the  user  commun- 
ity and  provide  a  basis  for  the 
installation  manager  to  understand 
more  clearly  their  needs.  Satisfac- 
tory performance  at  each  service  level 
is  defined  in  terms  of  criteria  that 
must  be  met  by  a  specified  percentage 
of  jobs  in  each  class. 

2.2.3  Measuring  Service 

The  measurement  of  service  levels 
depends  on  both  the  categories  of  work 
and  the  types  of  service  which  the 
users  demand.     For  example,  measures 
of  timeliness  for  batch  jobs  could  be 
expressed  as  the  number  of  jobs  on- 
time  (or  late) ,  average  turnaround 
time,  or  average  throughput.  Measures 
of  timeliness  for  interactive  process- 
ing normally  include  the  average  times 
for  responses,  as  well  as  the  consist- 
ency of  the  responses  for  similar 
tasks   [see  FIPS  57].     Availability  is 
measured  for  the  total  system,  as  well 
as  for  each  subsystem.  Reliability 
encompasses  both  hardware  components 
and  software  modules.  Finally, 
revenue  data  is  usually  a  function  of 
and  obtained  from  the  particular 
chargeback  system  being  used. 

Much  of  the  batch  service  level 
measurement  data  may  be  collected  from 
the  accounting  logs  found  on  most 
vendor  systems.     However,  most  vendor 
systems  do  not  provide  the  necessary 
data  for  interactive  or  remote  batch 
measures.     Special  hardware  devices, 
or  modifications  to  the  system  soft- 
ware, may  be  necessary  in  these  cases. 
Depending  on  the  definitions  of  the 
service  measures,  manual  logging 
procedures  may  even  be  necessary. 


2.2,4  Reporting  to  Users 

Many  installations  overlook  the 
benefits  that  can  be  derived  from 
organized  reporting  to  the  user 
community.     Proper  publicity  can  help 
both  to  accurately  report  on  the 
service  the  users  are  receiving  and  to 
influence  their  behavior.     The  atti- 
tude of  the  user  community  toward  the 
computer  installation  is  a  determining 
factor  in  the  effectiveness  of  the 
installation.     The  installation  may 
not  be  meeting   its  service  level 
objectives,  but  may  be  performing  its 
functions  in  the  best  way  possible. 
That  is,  there  are  times  when  the 
causes  of  poor  performance  may 
actually  lie  in  the  practices  of  the 
users,  and  changes  in  these  practices 
are  needed  to  improve  overall  instal- 
lation performance.     However,  dissat- 
isfied users  are  often  unlikely  to  be 
cooperative  in  making  changes  which 
would  improve  the  efficiency  and 
effectiveness  of  the  facility  as  a 
whole . 

Reports  to  the  user  community 
should  indicate  how  effectively  the 
installation  has  been  meeting  the 
service  requirements  of  the  user  and 
what  he  can  expect  to  receive  in  the 
future.     Users  are  generally  more 
interested  in  the  number  of  reports 
which  are  on-time,  the  batch  turn- 
around time,  or  the  timesharing 
response  time  than  in  CPU  utilization 
or  any  other  measures  of  hardware 
performance.     Other  measures  of 
effectiveness  encompass  the  areas  of 
reruns  and  aborts,  user  complaints, 
and  the  availability  and  reliability 
of  the  system  as  seen  by  the  various 
user  groups  or  classes  of  jobs.     It  is 
important  that  the  definitions  of 
turnaround  time,  availability,  etc. 
be  well-defined y  well-understood  and 
mutually  acceptable  to  both  the  ADP 
manager  and  the  user  community. 
Examples  have  been  given  in  the 
literature  showing  the  wide  dispari- 
ties between  how  the  user  perceives 
(defines)    a  measure  and  what  is 
actually  measured  by  the  installation 
(for  example,  see   [STEVD  781). 

The  reports  to  the  users  should 
display  more  than  averages  since  a 
particular  user's  job  is  rarely  "the 
average  job,"   Reports  showing  distri- 
butions for  the  various  measures  can 
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demonstrate  to  the  user  what  service 
he  can  expect  to  receive  most  of  the 
time  and  that  occasionally  things  are 
not  as  good  as  they  should  be.  Of 
course,  the  reports  may  only  serve  to 
show  him  that  service  is  not  as  good 
as  he  would  like  and  that  he  should 
alter  his  expectations  accordingly  or 
seek  to  renegotiate  the  service  levels 
and/or  the  cost  for  service.  Reports 
displaying  service  measures  by  class 
over  time  can  indicate  to  the  user 
what  to  expect  if  he  begins  a  differ- 
ent type  of  processing  or  if  he 
changes  the  characteristics  of  his 
jobs.     Reports  by  time  of  day,  time  of 
month,  or  some  other  cyclical  period 
may  indicate  to  the  user  that  he  may 
obtain  better  service  by  performing 
his  work  in  the  off-peak  periods. 

The  reliability  information  used 
in  vendor  relations,  although  usually 
reported  to  the  users  in  terms  of 
availability,   is  also  useful  to  the 
ADP  manager  in  assessing  the  quality 
of  service  he  is  providing  the  user 
community.     An  unreliable  system 
translates  into  a  system  which  is 
unavailable  when  the  user  needs  it  and 
into  a  system  which  causes  the  user  to 
have  to  repeat  lost  work.     It  is  quite 
possible  that  an  unreliable  and 
sporadically  available  system  causes 
more  user  frustration  and  hostility 
than  a  slow  one.     At  least  the  user 
knows  what  to  expect  from  a  system 
giving  poor  service. 

2.2.5  Correcting  Poor  Service 

Poor  service  may  be  caused  by  a 
true  lack  of  hardware  capacity  or  may 
be  the  result  of  poor  utilization  of 
existing  resources.     Some  remedies  for 
poor  utilization  fall  under  resource 
management   (see  Section  2.3)  but 
others  lie  in  changing  the  manner  in 
which  the  user  community  uses  the 
installation.     With  respect  to  the 
users,   increasing  capacity  may  be 
accomplished  by  influencing  the  user 
community  to  limit  their  resource 
usage  through  charging  practices,  by 
workload  management,  or  by  increasing 
user  proficiency.     Each  of  these 
possible  remedies  is  discussed  below. 

A  method  often  used  by  ADP 
managers  to  limit  resource  usage  is 
charging  the  users  for  the  resources 
and  services  they  receive.  Without 


some  type  of  fiscal  responsibility, 
the  workload  at  an  installation  may 
grow  in  an  uncontrollable  manner. 
Chargeback  is  more  effective  when  the 
user  of  the  installation  computer  and 
the  person  paying  the  bill  are  very 
closely  associated.     In  installations 
where  the  user  has  little  fiscal 
responsibility,  or  the  budget  cycle 
does  not  encourage  economy  of  resour- 
ces, the  chargeback  method  alone 
produces  little  change  in  user 
behavior.     However,  charging  users  for 
their  jobs  is  still  often  quite  an 
effective  means  of  controlling  unrea- 
sonable requests  and  preventing 
uncontrolled  growth.     The  charges 
should  be  directly  related  to  the 
actual  resource  utilization  and  should 
be  understandable  to  the  user  so  that 
he  can  control  his  usage  of  resources. 
Further,  the  charges  should  be  approx- 
imately repeatable  when  the  identical 
job  is  run  again  on  the  system.  Cost 
should  be  linked  to  the  impact  the  job 
has  on  the  installation's  performance. 
The  user  who  chooses  resource  demand- 
ing, high  priority  processing  should 
be  charged  accordingly.     Perhaps,  such 
a  user  should  be  notified  what  extra 
cost  he  is  incurring  through  his 
current  mode  of  operations  as  compared 
to  alternate  methods,  priorities,  or 
times  of  execution.     The  chargeback 
system  should  provide  economic  incen- 
tives for  the  users  to  request  only 
cost  effective  services. 

Workload  management  is  a  tool  of 
the  ADP  manager  which  is  not  used 
often,  although  it  is  well  known  that 
great  gains  can  be  made  by  eliminating 
little-needed  work.     In  times  of  low 
usage  and  great  capacity  at  an  instal- 
lation, the  problem  of  resources 
consumed  by  non-essential  workload  is 
not  so  crucial.     But,  as  excess 
capacity  begins  to  dwindle,  an  audit 
of  the  existing  workload  should  be 
performed  and  definite  steps  taken  to 
eliminate  unnecessary,  and  perhaps 
even  marginal,  workload  from  the 
computer.     In  reality,  the  audit  of 
the  existing  workload  for  the  instal- 
lation should  be  an  on-going  project. 
Not  only  can  work  of  limited  worth  be 
kept  off  the  machine,  but  workload 
statistics  can  highlight  the  top  users 
of  resources  who  may  then  receive 
training  or  guidance  by  the  ADP 
department  to  enhance  their  efficiency 
in  using  the  system. 
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Another  way  to  reduce  resource 
usage  is  by  upgrading  the  proficiency 
of  the  user  community  through  training 
programs,  meetings,  newsletters,  tech- 
nical support,  documentation,  etc. 
General  areas  for  improvement  can  be 
determined  by  the  examination  of 
processing  errors  and  their  causes. 
If  errors  are  further  categorized  by 
application  or  user,  customized 
approaches  can  be  used  for  the  top 
offenders.     If  installation  personnel 
are  available,  they  should  work  with 
the  users  during  their  planning  and 
design  phases  to  provide  guidance  to 
better  approaches  to  the  solution.  It 
is  often  beneficial  to  optimize  the 
large  production  programs.     Tools  and 
package?  exist  to  aid  the  installation 
in  this  task  once  problem  programs 
have  been  identified.     It  should  be 
noted  that  it  is  better  and  easier  to 
optimize  programs  during  the  design 
phase  than  after  development. 


2.3  Resource  Management 

A  systematic  approach  to  resource 
management  requires  a  non-trivial 
amount  of  time,  effort,  and  money  but 
may  provide  for  better  operations  in  a 
better  accepted   installation.  Effi- 
cient operation  of  the  installation  is 
needed   if  the  installation  is  to 
operate  at  least  cost.  Operational 
standards  need  to  be  set  and  tracked. 
Standards  for  hardware  utilization 
need  to  be  defined,  monitored,  and 
corrective  action  needs  to  be  taken 
when  necessary.     Forecasting  is 
required  to  ensure  the  continuation  of 
good  service  in  the  future.  Perform- 
ance measurement  for  resource  manage- 
ment may  begin  with  observation  and 
simple  manual  techniques  and  extend 
through  use  of  sophisticated  software 
and/or  hardware  monitors.     The  size 
and  maturity  of  the  installation 
should  determine  the  degree  of 
sophistication  of  the  performance 
measurement  program. 

2.3.1  Operational  Standards 

Operational  standards  and  pro- 
cedures should  be  defined  since  with- 
out approved  standards  and  procedures 
there  is  no  basis  for  performance 
evaluation  in  the  short  term.  They 
also  provide  the  basis  for  long-range 


planning  needed  to  meet  new  service 
criteria  or  simply  to  continue  meeting 
the  same  service  levels  in  an  expand- 
ing workload  environment.  Operational 
standards  might  encompass  such  func- 
tions as  reading  cards,  bursting  out- 
put, handling  media,  such  as  mounting 
tapes,  security  measures,  etc.  Proce- 
dural standards  would  define  proce- 
dures to  follow  for  unscheduled 
restarts,  cold  starts,  malfunctioning 
units,  etc.     These  standards  and 
procedures  must  not  be  rigid  but 
should  be  adaptable  to  technological 
and  environmental  changes  as  they 
occur.     For  a  rather  comprehensive 
document  on  the  subject,  the  reader  is 
referred  to  Standards  in  Operations 
[RICKJ  75] . 

Operational  procedures  will  work 
only  as  well  as  the  people  performing 
them.    Without  well-trained  and 
motivated  personnel,  the  effectiveness 
of  the  installation  can  be  seriously 
degraded.     The  ADP  manager  should  take 
steps  to  ensure  that  morale  is  and 
remains  good  at  his  installation.  Low 
morale  often  leads  to  sloppy  work 
which  will  show  up  as  operator  errors 
requiring  reruns,  delays  in  bursting 
output  that  inhibit  meeting  timeliness 
criteria,  etc. 

In  order  to  compare  the  actual 
performance  of  the  operational  proce- 
dures to  the  standards,  special  log- 
ging procedures  may  need  to  be  imple- 
mented in  order  to  capture  the  exact 
information  specified  by  the  service 
level  definitions.     For  example,  the 
user  may  require  turnaround  time  for 
local  batch  jobs  to  be  defined  as 
counter-to-bin  time  instead  of  the 
more  prevalent  and  easily  captured 
measure  of  cards-read  to  lines- 
printed.     Typically  in  the  operations 
spectrum,  work  flow  studies  are 
performed  to  measure  times  and  volumes 
of  work  in  critical  areas  such  as 
dispatching.     Another  source  of  data 
is  time  records  and  the  manual  logging 
of  events.     As  in  hardware  management, 
such  historical  data  can  then  be  used 
to  set  realistic  goals  and  can  also  be 
used  in  planning  by  comparing  future 
requirements  to  available  personnel 
hours.     Logs  of  errors  and  reruns  can 
be  studied  to  detect  areas  which 
require  further  operator  training  or  a 
modification  of  software  or  procedures 
for  easier  use  by  the  operators. 
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2.3.2  Resource  Utilization 
Standards 

Utilization  measurements  make  up 
the  primary  data  used   in  resource 
management.     Utilization  levels  by 
themselves  are  not  a  valid  measure  of 
installation  performance  but  are 
important  only  as  they  affect  instal- 
lation performance  criteria,  such  as 
service  to  the  user.     High  utilization 
levels  may  indicate  congestion  and 
consequent  system  degradation,  but 
congestion  and  degradation  are  not 
necessarily  related.     It  should  be 
remembered  that  utilization  levels  are 
not  only  a  function  of  internal 
computer  system  characteristics  and 
configuration  but  are  also  strongly 
influenced  by  external  factors  such  as 
scheduling  policies,  priority  process- 
ing requirements,  etc.     The  effects 
that  changes  in  configuration,  work- 
load and  application,  or  operating 
system  have  on  utilization  of  compon- 
ents and  subsystems  should  be  examined 
for  possible  causal  relationships. 
Information  on  such  relationships 
could  be  used  both  in  analysis  and 
tuning   in  the  operational  environment 
and  in  the  planning  process.  The 
inverse  process  of  examining  changes 
in  trends  of  utilization  to  ascertain 
changes  in  the  external  environment  is 
also  helpful   in  the  planning  process 
and  for  optimizing  operational 
procedures , 

Bell   [BELLT  75b]    introduced  the 
concept  of  using  control  limits  to 
help  the  ADP  manager  to  allocate  and 
plan  for  computer  resources  by  track- 
ing resource  consumption  to  know  when 
to  act  and  what  action  to  take. 
Simply  stated,  a  control  limit  is  a 
value  of  a  variable  which  should  not 
be  consistently  exceeded  if  the 
installation  is  to  continue  to  operate 
efficiently.     If  the  control  limit  is 
exceeded  some  action  should  be  taken 
to  return  the  variable  to  its  proper 
range;     i.e.,  the  controlled  area. 
Control  limits  for  hardware  units  and 
subsystems  can  be  developed  by  deter- 
mining the  values  at  which  the  users 
begin  to  complain,  by  plotting  service 
levels  versus  component  utilizations, 
etc.     Both  resource  usage  and  service 
levels  must  be  tracked  to  ensure  the 
acceptability  of  installation  perform- 
ance.    When  an  installation  is  not 
meeting  a  particular  service  level. 


there  can  be  many  causes.  By  looking 
at  resource  consumption  relative  to 
the  control  limits  set  to  keep  service 
levels  under  control,  the  ADP  manager 
can  anticipate  what  corrective  action 
to  take  and  when  to  take  it  to  ensure 
continuation  of  proper  service. 

Once  control  limits  are  set  there 
is  a  need  for  a  consistent  set  of 
management  policies  for  both  control 
limits  and  service  levels.     The  level 
of  response  will  not  necessarily  be 
the  same  for  all  control  limit  vio- 
lations but  should  be  commensurate 
with  the  significance  of  the  vio- 
lation.    The  ADP  manager  must  ensure 
that  enough  time  is  allowed  for  the 
evaluation  of  variances  and  that 
corrective  action  is  undertaken  when 
there  really  is  a  consistent  problem, 
but  that  no  action  is  taken  for  random 
occurrences.     The  use  of  control 
limits  can  reduce  costs  due  to  exces- 
sive capacity  and  can  enhance  the 
ability  to  consistently  provide  good 
service  by  reducing  the  need  for  fire- 
fighting  and  hasty  subjective 
j  udgments . 

2.3.3  Forecasting 

Forecasting  should  be  done  by  the 
ADP  manager  so  that  his  installation 
remains  responsive  as  new  work  is 
added,  the  existing  workload  changes, 
or  when  changes  in  installation  con- 
figuration could  result  in  less  cost. 
The  goals  of  forecasting  are  to 
predict  future  workloads  and  their 
resource  consumption;     to  anticipate 
problem  areas,  such  as  bottlenecks,  in 
time  to  prevent  them  from  occurring; 
and  to  predict  performance  changes 
that  may  result  from  changes  in  system 
configuration  and  workload.     With  the 
long  procurement  times  in  the  Federal 
sector  and  difficulties  in  enhancing 
existing  systems  speedily,  forecasting 
accurately  over  a  several  year  span  is 
necessary.     However,  even  when  the  ADP 
manager  can  accurately  forecast  the 
future  workload  for  his  installation, 
he  may  be  faced  with  the  problem  of 
forecasting   in  the  face  of  a  rapidly 
changing  technology,  which  may  affect 
the  manner  of  processsing   in  the 
future.     Examples  in  recent  years 
include  the  shift  from  batch  to  online 
processing  and  the  introduction  of 
data  base  management  systems. 
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The  process  of  forecasting  can 
not  be  performed  by  the  ADP  manager  in 
isolation  but  must  involve  the  users 
who  are  the  source  of  information  on 
the  characteristics  of  new  workload, 
new  service  requirements,  or  the 
expansion  or  contraction  of  existing 
workload.     The  obvious  patterns  in 
workload  can  be  predicted  from  trend 
data,  but  the  effect  from  the  intro- 
duction of  new  applications,  etc. 
must  also  be  estimated  and  such  in- 
formation comes  from  the  user  commun- 
ity.   The  workload  must  be  described 
in  terms  which  can  be  understood  by 
the  user  and  can  be  used  as  the  basis 
for  the  user  to  project  his  future 
requirements.     But,  the  user  descrip- 
tion must  also  be  translatable  into 
terms  used  by  the  forecasting  tech- 
nique.    The  process  of  forecasting  may 
also  involve  upper  management, 
particularly  if  the  installation  must 
acquire  capacity  for  future  use  and, 
hence,  may  require  an  interim  subsidy. 

The  forecasting  process  begins 
with  the  characterization  of  the 
current  and  projected  workload.  There 
are  many  techniques  for  characterizing 
workloads  which  range  from  simple  job 
stream  analysis  to  determine  the 
average  job  characteristics  to  sophis- 
ticated clustering  algorithms  which 
statistically  categorize  job  classes. 
The  degree  to  which  the  workload  can 
be  accurately  described  strongly 
influences  the  accuracy  of  any  fore- 
casting done  by  the  ADP  manager. 
However,  sometimes  the  ADP  manager 
needs  only  rough  estimations  and  the 
simpler  techniques  suffice  in  such 
instances . 

Job  stream  profiles  are  typically 
graphs  displaying  percentage  of  jobs 
versus:     CPU  time  per  job,  number  of 
tapes  per  job,  elapsed  time  per  job, 
memory  used  or  required  per  job,  etc. 
The  graphs  provide  a  visual  means  for 
the  ADP  manager  to  partition  his 
installation's  workload.     Job  stream 
profiles  also  provide  rough  esti- 
mations of  workload  changes  and  can 
indicate  when  the  composition  of  the 
workload  is  staying  steady.     In  most 
installations,  whether  production  or 
not,  the  composition  of  the  workload 
will  remain  steady  for  substantial 
periods  of  time.     Hence,  the  average 
job,  or  jobs,  for  the  installation  may 
be  defined  by  resource  consumption 


characteristics.     For  example,  the 
average  compile-only  job  may  use  two 
seconds  of  CPU  time  and  require  twenty 
thousand  units  of  memory.     These  aver- 
age jobs  can  be  used  then  as  coarse 
measures  of  workload  when  measuring 
system  productivity  or  doing  "ball 
park"  forecasting  of  workload.  Work- 
load statistics  of  this  type  can  point 
to  growth  in  specific  applications  or 
changes  in  the  workload  composition. 
Continuing  workload  monitoring  can 
provide  data  to  support  benchmarking 
and  simulation  work  used  in  the 
planning  process.     More  sophisticated 
means  of  workload  characterization  — 
for  example,  clustering  —  also  exist 
(for  example,  see   [AGRAA  78],  [DEVEE 
78],    [FERRD  72],    [MEADR  78]). 

The  reason  for  forecasting 
usually  determines  the  specific 
techniques  which  would  be  used  (for 
example,  see   [ABERF  78],    [AGRAA  78], 
[CANNR  75],    [GILMM  78],    [HERNE  76], 
[MEEHJ  78]).     The  specific  techniques 
include  benchmarking,  regression 
analysis,  time  series  analysis, 
response  surfaces,  analytical  model- 
ling, operational  analysis,  and  many 
more.     Of  course,  the  workload 
characterization  technique  must 
provide  the  proper  type  of  data  for 
the  forecasting  technique  being  used. 
Unfortunately,  some  of  the  forecasting 
techniques  are  difficult  to  use  or  are 
inaccurate.     Further  clouding  the 
issue  is  the  question  of  how  to  fore- 
cast user  adaptability  to  future 
changes  in  the  system.     Despite  these 
problems,  forecasting  at  some  level  of 
detail  and  accuracy  must  still  be 
done . 


2.4  Vendor 

During  the  operational  phase  of 
the  ADP  life-cycle,  the  ADP  manager 
must  ensure  that  the  vendor  lives  up 
to  his  obligations  under  the  contract. 
Furthermore,  the  manager  may  need  to 
enlist  the  vendor's  support  in  the 
correction  of  persistent  or  increasing 
problems  with  system  failure.  The 
collection  and  review  of  concrete 
reliability  data  will  foster  mutual 
understanding  of  problems  and  cooper- 
ation between  the  ADP  manager  and  the 
vendor  in  finding  solutions.    When  the 
ADP  manager  must  rely  on  supposition 
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and  intuition  for  his  arguments, 
antagonistic  and  non-cooperative 
relationships  may  evolve  with  the 
vendor . 

When  a  system  malfunctions,  the 
problem  is  usually  the  result  of  hard- 
ware or  software  malfunction,  but 
finding  the  culprit  is  not  always  a 
simple  matter.     The  problem  is  diffi- 
cult even  when  the  installation  has  a 
single  vendor  supplying  both  the 
software  and  the  hardware.     In  multi- 
vendor  environments,  the  problem  of 
assigning  responsibility  for  the 
malfunction  is  compounded  and  the  need 
for  accurate  information  on  the 
reliablity  of  each  module  or  device  is 
necessary  for  both  pinpointing  the 
specific  problem  and  for  ensuring 
solutions  to  recurring  problems. 

Information  on  error  rates  by 
device  or  software  module  should  be 
collected,  as  well  as  information  on 
downtimes  for  preventive  (scheduled) 
maintenances  and  the  frequency  and 
duration  of  remedial  (unscheduled) 
maintenance.     Displaying  this  data  as 
trends  and  peaks  can  assist  the  ADP 
manager  in  detecting  declining  per- 
formance and  in  identifying  inadequate 
vendor  support  areas.     Collection  of 
reliability  data  may  also  allow  the 
installation  manager  to  make  compar- 
isons with  installations  of  similar 
environment  and  configuration. 

Time  lost  due  to  a  computer 
system  malfunction  translates  into 
loss  of  "availabil ity  to  the  user 
community.     The  modularity  of  many 
computers  today  clouds  the  issue  of 
availability  since  many  systems  can 
continue  to  operate  in  degraded  mode. 
For  example,  the  system  may  operate 
more  slowly  when  a  device  is  down  and 
similar  devices  pick  up  the  work  for 
the  malfunctioning  unit.     In  other 
cases,  jobs  may  not  be  scheduled  which 
require  devices  for  which  there  are  no 
backup  capabilities.     Evaluation  of 
availability  of  such  systems  can  be 
extremely  complex  especially  in  the 
latter  case  where  the  system  could  be 
described  as  being  "partially  avail- 
able;" i.e.,  it  is  unavailable  for  the 
user  class  requiring  the  malfunction- 
ing device  but  available  to  all  other 
users . 


2.5  Summary 

The  sections  above  discussed  many 
functions  which  the  ADP  manager  should 
perform  in  fulfilling  the  goal  of 
providing  good  service  to  the  users  at 
the  least  cost.     The  upper  management 
and  user  relationships  are  extremely 
important  to  the  proper  management  of 
the  ADP  installation.     The  ADP  manager 
cannot  plan  well  and  provide  the 
necessary  service  unless  he  has  strong 
relationships  in  both  of  these  two 
areas.     In  an  installation  where  both 
the  users  and  the  ADP  installation  are 
under  the  same  upper  management 
structure  the  ADP  manager  should  be 
able  to  exert  strong   influence  on  his 
user  community  and  vice  versa.  These 
relationships  are  vital  to  the  plan- 
ning process  if  the  needed  installa- 
tion capacity  is  to  be  available  when 
new  work  arrives.     When  the  users' 
upper  management  is  not  the  same  as 
that  of  the  ADP  installation,  the  user 
relationship  becoihes  even  more  impor- 
tant in  the  areas  of  planning  and 
workload  management. 

To  support  the  ADP  manager  in 
performing  the  tasks  for  all  four 
functional  areas,  phe  CPM  program  must 
supply  him  with  the  proper  informa- 
tion.    The  data  items  making  up  this 
information  come  from  many  sources  — 
accounting  logs,  manual  logs,  finan- 
cial statements,  hardware  and  software, 
monitors,  surveys,  etc.     A  single  data 
item  —  for  example,  CPU  time  —  may 
also  support  several  informational 
needs  and  be  used  in  multiple  ways. 
Appendix  A  presents  a  model  depicting 
the  informational  needs  of  the  ADP 
manager.     This  model  was  developed  by 
expanding  the  functions  described 
above  into  the  classes  of  information 
needed  to  support  each  function. 
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3.0  Installation  Performance 
Activities  in  the  Federal 
Sector 

The  functions  described   in  Sec- 
tion 2  above  constitute  best  practices 
for  managing   installation  performance. 
These  functions  define  a  baseline  for 
comparison  with  common  practices  in 
the  Federal  sector.     As  a  result  of 
previous  communication  with  personnel 
at  Federal  and  private  sector  ADP 
installations,  attendance  at  computer 
performance  conferences,  and  discus- 
sions with  computer  performance 
consultants,  certain  hypotheses  were 
formed  concerning  the  current  state  of 
installation  performance  and  measure- 
ment activities  in  the  Federal  and 
private  sectors.     In  order  to  verify 
these  hypotheses,  site  visits  were 
made  to  nine  Federal  ADP  installations 
and  the  various  computer  performance 
auditing  organizations  within  the 
Federal  government.     At  the  installa- 
tions, the  discussion  included  the 
question  of  how  the  installation 
evaluates  itself,  what  information  is 
used  or  needed  by  the  ADP  manager  and, 
if  more  performance  activities  could 
be  done  at  their  site,  what  prevents 
them  from  being  attempted.     In  the 
interviews  with  the  auditing  agencies, 
the  topics  of  discussion  were  what 
types  of  and  how  many  audit  functions 
they  perform,  the  state  of  installa- 
tion performance  from  their  perspec- 
tive, and  what  could  be  done  to 
improve  the  situation.     The  views  on 
CPM  practices  expressed  by  the  Federal 
ADP  performance  auditor  and  the  ADP 
installation  managers  will  be  dis- 
cussed  in  turn  below.     The  perceptions 
of  the  interviewees  concerning  the 
reasons  why  more  activity  is  not 
present  are  included   in  Section  4. 

We  do  not  directly  address  the 
question  of  how  the  Federal  sector 
compares  with  private  industry.  The 
situation  with  respect  to  installation 
performance  work  appears  to  be  similar 
in  both  sectors  although  the  emphasis 
may  be  different.     For  example,  the 
private  sector  may  not  have  the  re- 
quirement for  competitive  procurement 
or  upgrade  and,  hence,  may  limit  its 
planning  and  forecasting  to  various 
machine  sizes  within  a  single  vendor 
line. 


3.1  Auditor  Perspective 

In  order  to  verify  the  hypotheses 
which  were  made  concerning  the  current 
state  of  installation  performance  ac- 
tivities,  it  was  decided  that  computer 
performance  audit  groups  within  the 
Federal  government  should  be  inter- 
viewed.    Each  of  these  agencies  does 
performance  audits  at  many  Federal  ADP 
sites  each  year.     Hence,  they  would  be 
quite  knowledgeable  about  the  status 
of  CPM  activities.     Below  we  briefly 
discuss  the  activities  of  the  agen- 
cies which  were  interviewed  and  sum- 
marize the  remarks  and  points  which 
they  made  concerning  their  view  of 
installation  performance  activities  in 
the  Federal  government. 

Six  individuals  from  all  four  of 
the  known  performance  auditing  agen- 
cies were  interviewed.     One  of  the 
agencies  is  a  true  audit  agency  and 
the  others  do  performance  audits  in 
conjunction  with  their  role  as  invited 
advisors  on  installation  performance 
problems.     The  first  agency,  the  Gen- 
eral Accounting  Office   (GAO)  ,  performs 
audits  to  establish  how  much  capacity 
exists  at  an  installation  and  how  much 
of  that  capacity  is  being  used.  The 
major  thrust  of  their  effort  is  to 
encourage  installations  to  plan  more 
effectively.     The  other  agencies  see 
as  their  goal  the  encouragement  of 
good  computer  management  practices  at 
installations.     All  of  these  latter 
agencies  are  a  part  of  the  Department 
of  Defense   (DOD) ,  although  one  serves 
more  than  DOD  installations. 

Each  of  the  agencies  had  reviewed 
dozens  of  sites  but  from  different 
perspectives.     The  less  experienced 
agencies  concerned  themselves  primar- 
ily with  utilization  measurements 
while  the  more  experienced  agency  had 
moved  from  an  emphasis  on  utilization 
statistics  to  the  need  for  measurement 
of  user  attitudes  as  a  starting  point 
in  the  performance  analysis  process. 

The  problems  with  installation 
performance  activities  which  were  most 
often  cited  during  the  interviews 
were:     (1)    failure  to  assign  personnel 
and  to  allocate  funds  to  a  CPM  pro- 
gram,  (2)   lack  of  upper  management 
support,    (3)    failure  of  ADP  managers 
to  consider  service  levels  and  their 
attainment  as  one  of  the  primary 
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measures  of  installation  performance, 
(4)   failure  to  institute  a  chargeback 
system  as  a  means  of  influencing  user 
behavior,  and   (5)   a  general  lack  of 
knowledge  and  training   in  CPM.  Each 
of  these  problem  areas  is  discussed  in 
more  detail  below. 

The  agencies  indicated  that  one 
of  their  primary  objectives  was  to 
convince  the  ADP  manager  that  CPM  is 
necessary  and  useful  and  that  CPM  can 
supply  information  to  support  more  of 
his  functions  than  is  commonly 
thought.     Few  of  the  sites  reviewed  by 
the  agencies  had  comprehensive  CPM 
programs  as  outlined  in  Section  2 
above.     The  initial  entry  into  a  CPM 
program  was  indicated  to  be  the  tuning 
efforts  of  the  installation.  The 
sites  mentioned  as  having  good  CPM 
programs  tended  to  use  a  mix  of  com- 
mercial accounting  packages,  locally 
developed  monitors,  and  chargeback 
systems . 

One  agency  indicated  that  it 
rarely  found  sites  where  the  ADP 
manager  regularly  reported  to  his 
upper  management.     And  at  the  sites 
which  did  report,  the  reports  them- 
selves were  usually  not  appropriate 
but  were  composites  or  summaries  of 
resource  management  reports.  They 
pointed  out  that  the  lack  of  reporting 
to  and  the  observed  lack  of  support  by 
upper  management  are  probably  related. 

The  agencies  noted  that  a  dif- 
ference in  installations  having  good 
CPM  programs  and  in  sites  having  poor 
CPM  programs  could  be  observed   in  the 
attitudes  of  the  installation  mana- 
gers.    At  the  good  sites  there  appear- 
ed to  be  a  genuine  concern  about  meet- 
ing service  levels.     Those  installa- 
tions viewed  their  mission  as  one  of 
providing  service  which  must  be. 
measured  and  planned  for  in  order  to 
meet  realistic  levels  of  service.  At 
other  installations,  the  agencies 
perceived  several  reasons  for  the 
failure  of  ADP  managers  to  consider 
service  levels  with  respect  to 
installation  performance:     (1)  poor 
user  relations  or  lack  of  concern  for 
the  users  by  the  ADP  manager,    (2)  ne- 
gative incentives  for  the  ADP  manager, 
(3)   lack  of  appropriate  measuring 
tools,  and   (4)    lack  of  adequate  fore- 
casting tools.     In  most  such  sites, 
poor  user  relations  seem  to  arise  from 


the  lack  of  both  service  goals  or 
standards  of  performance  and  well- 
defined  user  reporting  procedures. 
With  no  performance  goals  the  ADP 
manager  has  no  means  of  evaluating 
user  complaints  and  with  no  reporting 
system  the  users  tend  to  complain 
frequently.     Eventually  the  ADP 
manager  begins  to  believe  that  his 
installation  is  performing  poorly  and 
this  attitude  becomes  a  negative 
incentive  for  establishing  a  CPM 
program  for  fear  that  it  will  reflect 
poorly  on  his  work  as  a  manager.  When 
early  attempts  at  measuring  or  fore- 
casting are  inaccurate,  the  fear  of 
failing  again  leads  the  ADP  manager  to 
search  for  infallible  tools.  Failing 
in  his  search,  the  ADP  manager  moves 
further  away  from  implementing  a  CPM 
program  that  could  help  him  solve 
perceived  installation  performance 
problems . 

Chargeback  ijs  viewed  by  the  agen- 
cies as  more  than  simple  cost  recov- 
ery.    They  view  it  as  another  measure 
of  accountability  and  a  means  of  con- 
trolling  user  behavior.     The  key 
elements  were  defined  to  be  high  visi- 
bility among  the  users  --  not  just 
their  managers  --  coupled  with  actual 
low-level  fiscal  responsibility. 
Under  these  circumstances  the  agencies 
believed  that  the  users  would  be 
encouraged  to  exercise  more  care  in 
the  development  and   implementation  of 
programs  and  would  be  more  likely  to 
demand  reasonable  service  and  resource 
levels.     Unfortunately,  they  reported 
finding  few  sites  with  good  chargeback 
systems . 

Lack  of  knowledge  and  training 
had  been  observed  by  the  agencies  with 
respect  to  both  the  user  and  the  ADP 
installation  personnel.     On  the  user 
side  are  found  users  who  are  not  aware 
of  efficient  ways  to  program  nor 
otherwise  utilize  the  full  capabil- 
ities of  the  system.  Additionally 
they  are  rarely  aware  of  the  problems 
of  the  ADP  manager  or  the  demands 
being  made  by  other  users  which  could 
affect  their  own  use  of  the  installa- 
tion.    On  the  installation  side  are 
found  managers  who  often  lack  even  a 
rudimentary  knowledge  of  the  CPM 
process  or  its  benefits.     Lacking  such 
knowledge  and  facing  turnovers  and 
training  expenses,  the  ADP  manager  is 
hesitant  to  invest  the  necessary  funds 
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and  personnel   in  CPM  training. 


3.2  Installation  Perspective 

Although  the  NBS  staff  is  in 
frequent  contact  with  many  Federal  ADP 
installations,   it  was  decided  that  a 
formalized  interview  procedure  should 
be  undertaken  to  ascertain  the  types 
of  installation  performance  measure- 
ment and  reporting  procedures  which 
are  used   in  the  Federal  government. 
Site  visits  were  conducted  to  inter- 
view nine  ADP  managers,  or  the  CPM 
managers  if  the  ADP  manager  was 
unavailable,  for  installations  in  the 
greater  Washington  metropolitan  area. 
The  selection  criteria  was  not  formal 
but  attempted  to  include  sites  from 
many  of  the  major  Federal  departments, 
sites  that  supported  central  agency 
functions,  sites  that  are  "field 
installations,"   sites  that  were  known 
to  have  at  least  some  computer  per- 
formance program,  and  sites  about 
which  nothing  was  known  concerning 
their  installation  performance  work. 

The  sites  visited  included 
installations  from  the  Departments  of 
Defense,  Agriculture,  Commerce,  and 
Health,  Education  and  Welfare.  In 
addition,  information  from  a  previous- 
ly conducted  NBS  review  [CONTD  79]  of 
the  U.S.     Postal  Service  CPM  program 
was  used.     We  will  not  attempt  to  gen- 
eralize on  the  amount  of  CPM  work 
which  is  performed  at  Federal  sites 
throughout  the  country  but  will 
restrict  our  comments  to  what  we  found 
at  these  sites.     Of  the  nine  sites, 
three  could  be  characterized  as  having 
good,  though  not  complete,  programs; 
one  as  having  a  fair  CPM  program; 
three  as  having  poor  efforts;     and  two 
as  having  their  programs  in  various 
stages  of  development  and  specifi- 
cation, but  little  implementation. 

During  the  interviews  the  dis- 
cussion centered  on  the  folir  areas  of 
ADP  manager  responsibility  which  are 
given  in  the  model  above  (Figure  1). 
The  intent  was  to  find  out  what  the 
installation  was  currently  doing  in 
those  areas,  what  the  ADP  manager 
thought  they  should  be  doing   in  those 
areas,  and  what  types  of  information 
he  needed  to  perform  his  job.  It 
might  be  of  interest  to  note  that  the 


ADP  managers  of  sites  with  good  CPM 
programs  were  more  aware  of  the 
inadequacies  in  their  programs  than 
the  managers  whose  programs  were  some- 
what less  than  adequate.     In  fact, 
during  the  interviews  it  was  evident 
that  the  latter  managers  were  often 
unaware  of  other  practices  or 
functions  which  could  be  performed  or 
the  benefits  which  could  result  from 
such  practices. 

The  relation  to  upper  management 
was  found  to  be  the  least  developed  of 
all  the  areas.     Very  few  of  the  sites 
did  any  reporting  to  upper  management 
and  only  one  site  presented  inform- 
ation which  was  management  oriented. 
A  few  other  sites  sent  various  types 
of  utilization  reports  to  their  mana- 
gement.    Often  these  reports  were 
nothing  more  than  "management  reports" 
produced  from  commercially-available 
software  packages   (for  example, 
accounting  log  analyzers,  software 
monitors).     In  fact,  these  reports 
would  be  more  appropriately  classified 
as  condensed  resource  management 
reports.     At  several  sites  the  ADP 
manager  expressed  surprise  at  our 
inquiry  and  even  questioned  the  need 
for  such  a  relationship  with  upper 
manag  ement . 

The  user  relationship  area  was 
the  best  developed  area  and  all  of  the 
sites  had  some  type  of  user  community 
liaison  effort.     The  most  popular 
mechanisms  were  the  use  of  regular, 
usually  monthly,  meetings  of  user 
groups;     user  liaison  personnel; 
newsletters;     and  trouble  desks  for 
answering  questions.     Few  of  the  sites 
had  formal  user  reporting.  The 
installations  with  strong  user  support 
groups,  who  believed  their  function  to 
be  user  service,  seemed  to  have  over- 
come the  problem  of  user  dissatisfac- 
tion.    In  addition,  such  sites  usually 
had  good  user  training  programs.  Only 
one  site  actually  negotiated  service 
levels  and  charges  with  their  users. 
Few  of  the  sites  had  a  chargeback 
system  which  caused  funds  to  be 
transferred  from  the  user  project  to 
the  ADP  installation,  and  those 
chargeback  systems  were  oriented 
toward  cost  recovery  with  little 
emphasis  on  influencing  user  behavior. 
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All  of  the  sites  did  some  per- 
formance tuning   in  the  area  of  re- 
source management  but  many  were 
restricted  to  one-time,  isolated 
projects.     The  three  sites  with  good 
CPM  programs  all  kept  historical  data 
for  trends  and  analysis  and  to  help 
them  in  setting  control  limits.  One 
of  these  three,  however,  was  prohib- 
ited from  openly  setting  control 
limits  for  fear  that  the  installa- 
tion's performance  would  become  more 
visible.     The  other  six  sites  appeared 
not  to  use  control  limits  but  the  ADP 
managers  were  satisfied  with  a  fire- 
fighting  mode  of  operation.  Workload 
statistics  were  commonly  reported  to 
the  users  in  addition  to  some  inform- 
ation in  terms  of  responsiveness  and 
availability.     Capacity  planning  and 
forecasting  were  a  rare  occurrence 
and,  when  attempted  at  all,  were 
usually  restricted  to  projections 
based  on  the  utilization  of  one 
component  —  normally  the  CPU. 

In  the  area  of  vendor  relations 
the  sites  varied  according  to  the 
formality  of  the  arrangement  but  all 
did  some  tracking  of  equipment  reli- 
ability.    Several  sites  used  commer- 
cially-available packages  to  track  the 
performance  of  their  equipment  versus 
other  sites  with  the  same  equipment. 
Most  of  those  sites  believed  that  such 
information  gave  them  more  leverage 
with  the  vendor.     Most  of  the  sites 
used  their  trouble  reports  in  a 
fire-fighting  mode  —  if  a  problem 
seemed  to  persist,  the  ADP  manager 
went  to  the  files  and  looked  back 
through  the  paper  console  logs  and 
trouble  forms  for  any  trends.  One 
site  had  its  own  automated  system 
which  it  ran  regularly  to  try  to 
detect  any  equipment  which  might  be 
degrading  and  to  use  this  information 
in  its  dealings  with  its  vendors. 


the  important  upper  management  path  is 
quite  weak  while  the  strength  usually 
lies  in  the  user  community  relation- 
ship, although  this  area  could  also  be 
enhanced  with  better  reporting 
practices.     Dealing  with  the  vendors 
was  the  most  uniform  across  sites 
while  the  amount  of  resource 
management  performed  varied  the  most 
between  sites. 

As  is  noted  above,  it  does  not 
appear  that  the  Federal  government 
practices  are  any  better  or  worse  than 
would  be  found  in  the  private  sector. 
However,  there  would  appear  to  be 
ample  room  for  improvement  in  all  of 
the  four  areas  described   in  Section  2 
above.     In  Section  4  we  will  discuss 
possible  underlying  causes  for  these 
inadequac  ies . 


3.3  Summary 

None  of  the  findings  of  the 
interviews  were  surprising,  although 
they  tended  to  confirm  previous 
hypotheses  concerning  the  quantity  and 
quality  of  installation  performance 
work  being  done  in  the  Federal  govern- 
ment.    There  is  a  general  lack  of  CPM 
programs  in  the  Federal  sector.  At 
the  sites  doing  any  performance  work. 
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4.0  Current  Inadequacies  in 

Installation  Performance  Practices 

How  do  the  current  practices  in 
the  Federal  government  compare  to 
state-of-the-art  possibilities?  In 
Section  2  above  we  discussed  a  number 
of  functions  which  could  be  performed 
by  an  ADP  manager  in  relating  to  his 
upper  management,  to  users,  and  to 
vendors  and   in  managing  his  instal- 
lation resources.     Section  3  outlined 
the  practices  at  a  small  number  of 
Federal  sites  which  were  visited  and 
the  collaborating  views  expressed  by 
personnel   in  several  Federal  computer 
performance  auditing  agencies.  The 
site  visits  and  the  interviews  con- 
firmed the  hypotheses  which  had  been 
made  concerning  current  practices.  A 
gap  exists  between  what  is  practiced 
and  what  could  be  done.     In  partic- 
ular, the  upper  management  relation 
rarely  exists;     fire-fighting   is  the 
prevalent  mode  of  operation  instead  of 
managed  performance;     little  user 
reporting   is  done;     and  very  few  sites 
negotiate  service  levels. 

This  lack  of  wel 1 -developed  and 
widely-used  CPM  programs  is  the  result 
of  the  magnitude  of  the  effort  cur- 
rently required  to  have  a  CPM  program 
at  an  installation.     This  conclusion 
is  supported  by  the  literature,  by 
conversations  with  computer  perform- 
ance conference  attendees,  by  obser- 
vation, and  by  remarks  made  during  the 
site  visits.     CPM  programs  can  require 
substantial  personnel  time  to  develop 
and  operate.     The  nature  of  the  work 
further  requires  that  the  personnel 
who  do  the  work  be  experienced  and 
quite  knowledgeable  about  the  system 
being  measured.     In  this  section  we 
will  discuss  several  possible  under- 
lying  reasons  for  the  stringent  re- 
quirements of  installation  performance 
activities:      (1)   lack  of  measurabil- 
ity,    (2)   lack  of  well-defined  terms, 
(3)   lack  of  "good"  values,  and  (4) 
lack  of  statistical  Computer  Perfor- 
mance Evaluation   (CPE)  techniques. 
Section  5  w.i  1 1  outline  a  few  initial 
steps  which  could  be  undertaken  to 
alleviate  some  of  these  problems. 


4.1  Lack  of  Measurabil ity 

Before  the  ADP  manager  can  manage 
the  performance  of  his  installation, 
he  must  have  access  to  certain  inform- 
ation.    Unfortunately,  most  of  the 
computer  systems  in  use  today  were  not 
designed  to  be  measured  and  do  not 
supply  all  of  the  needed  information. 
The  common  measures  which  are  provided 
by  most  vendors  were  originally  for 
accounting  purposes  and  not  perform- 
ance management.     Additionally,  most 
vendors  supply  only  a  small  fraction 
of  the  information  needed  by  the  ADP 
manager  in  ensuring  good  installation 
per  formance . 

Many  of  the  measures  needed  in 
the  performance  area  are  not  readily 
accessible  but  are  buried  in  the 
system.     Hardware  and  software  moni- 
tors are  necessary  to  retrieve  the 
needed  information  but  the  monitors 
themselves  may  cause  degradation  of 
system  performance.     All  software 
monitors  consume  resources  to  a 
greater  or  lesser  degree  depending 
upon  the  architecture  of  the  system 
and  its  operating  system.     In  add- 
ition, the  use  of  hardware  monitors 
requires  specialized  expertise. 

Even  when  installations  go  to  the 
expense  and  time  of  installing  moni- 
tors, measurement  is  not  ensured. 
Some  measurements  are  prohibitively 
expensive  with  respect  to  the  substan- 
tial computer  resources  consumed 
during  the  measurement,  data  collec- 
tion, and  analysis  process.  Other 
measurements  are  not  possible  without 
substantial  system  alterations  and  may 
still  consume  many  computer  resources 
when  put  in  place. 

From  observation  and  from  the 
interviews,   it  would  appear  that  the 
lack  of  measurabil ity  contributes 
greatly  to  the  lack  of  performance 
programs  at  Federal  ADP  installations. 
Many  of  the  measures  are  not  directly 
provided  by  the  vendors  and  collecting 
and  analyzing  measurement  data  require 
the  time  and  effort  of  a  highly 
trained  staff.     Many  installations 
cannot  afford  the  drain  on  their 
resources  which  would  be  required  for 
an  installation  performance  program. 
Designing  system  hardware  and  software 
with  performance  measurement  in  mind 
and  liroviding  eas.y  access  to  the  data 
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would  reduce  the  effort  required  to  do 
performance  management  at  individual 
install  at ions . 


4.2  Lack  of  Well-Defined  Terms 

Stevens  [STEVB75b-pl]   summed  up 
the  terminology  problem  in  the  per- 
formance area  when  he  stated: 

As  an  industry  we  suffer  from  a 
complete  absence  of  both  standard 
units  of  measure  and  standard 
metrics.     With  standard  units  we 
could  make  direct  comparisons 
among  equipment  types  available 
to  service  the  same  workload,  and 
identify  the  precise  configu- 
ration having  the  absolute  mini- 
mum amount  of  capacity  necessary 
to  do  the  job.     ...  Failing 
that,  we  could  utilize  standard 
metrics  to  at  least  describe  to 
each  other  the  selection  and 
evaluation  processes  we  use  to 
assure  ourselves  that  we  are 
producing  the  best  possible 
result. 

The  lack  of  commonality  of  well- 
defined  terms  hinders  communication 
between  practitioners.     The  computer 
performance  field   is  one  of  the  few 
"professional   fields"  which  lacks 
standard  nomenclature  and,  as  a 
result,  has  probably  not  progressed  as 
rapidly  as  it  might  have.     It  is  not 
that  the  various  systems  collect 
vastly  different  information  which 
prevents  the  sharing  of  ideas  and 
techniques.     The  problem  lies  in  the 
fact  that  although  different  vendor 
systems  collect  essentially  the  same 
basic  set  of  data,  they  mean  slightly 
different  things  and  are  recorded  in 
different  formats.     Hence,  there  is 
little  completely  common  ground. 

Further  compounding  the  problem, 
is  the  lack  of  common  names  for  the 
same  measures.     Some  measures  from 
system  A  may  be  identical  to  those  on 
system  B,  although  different  names  are 
used.     A  performance  problem  on  system 
A  may  be  identical  to  a  performance 
problem  on  system  B  which  has  been 
solved  on  system  A.     But,  the  analysts 
on  system  B  do  not  recognize  that  the 
problems  are  the  same  because  the 
terms  are  different.     Hence,  they  can 


not  avail  themselves  of  an  approach  to 
the  solution  or  the  solution  itself. 

The  two  problems  —  measuring 
different  things  and  measuring  the 
same  thing  but  naming  it  differently 
—  combine  to  force  much  of  the  de- 
velopment work  in  CPM  to  be  repeated 
at  each  installation.     The  lack  of 
well-defined  and  common  terms 
seriously  limits  the  ability  of  small 
installations  with  limited  resources 
and  installations  new  to  CPM  to  use  an 
established  base  of  techniques, 
problem  approaches,  and  solutions 
developed  by  other  sites  and  verified 
by  repeated  use.     As  a  consequence, 
many  sites  do  not  use  CPM  to  as  great 
a  degree  as  they  could  or  they  do  not 
start  up  any  program  at  all.  The 
establishment  of  common  and  well- 
defined  measures  used  in  installation 
performance  work  would  also  allow 
greater  communication  among  instal- 
lations working  on  various  performance 
problems.     Common  terminology  would 
allow  work  to  begin  toward  establish- 
ing standards  of  "good"  performance 
and  ranges  of  normative  values  for 
operating  levels. 


4.3  Lack  of  "Good"  Values 

One  of  the  difficulties  of  an  ADP 
manager  is  knowing  what  constitutes 
"good"  performance.     When  a  system  is 
meeting  its  service  goals  and  no 
problems  are  visible  in  its  operation, 
how  can  the  ADP  manager  know  if  his 
system  is  well-tuned?     Before  this 
question  is  summarily  dismissed  as  not 
being  germane  to  installation  perform- 
ance, the  following  should  be  con- 
sidered.    An  installation  may  be  oper- 
ating close  to   its  performance  levels 
but  still  have  capacity  to  process 
additional  work.     When  new  work  is 
presented  to  an  installation,  the  ADP 
manager  needs  to  know  whether  suffi- 
cient excess  capacity  exists  in  his 
system  to  accommodate  the  new  work 
through  a  performance  enhancement 
effort  instead  of  an  upgrade  or  pro- 
curement.    Such  delays  of  upgrades  or 
procurements  could  save  the  installa- 
tion money.     The  arguments  that  the 
upgrade  would  be  cheaper  than  the 
performance  enhancement  effort  (since 
hardware  is  cheaper  than  personnel) 
indeed  have  merit  in  an  environment 
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where  procurements  or  upgrades  are 
easily  justified  and  do  not  take  long 
periods  of  time.     However,  such  an 
environment  is  rarely  found   in  the 
Federal  sector. 

Hence,  the  ability  to  compare 
installations   (i.e.,  to  determine  if  a 
particular  system  is  well-tuned)  could 
be  beneficial   in  the  Federal  ADP 
environment.     The  lack  of  industry- 
wide standards  of  "good"  performance 
even  prohibit  the  comparison  of  per- 
formance levels  among  installations 
with  like  configurations  and  workload. 
Such  standards  could  provide  a 
starting  point  for  setting  goals  for  a 
particular  installation  and  provide  a 
basis  for  the  ADP  manager  to  compare 
the  performance  of  his  installation 
with  respect  to  a  large  number  of 
other  "similar"   installations.  Such 
normative  values  could  assist  the  ADP 
manager   in  assessing  how  well  his 
system  is  operating  and  how  much 
capacity  could  be  obtained  for  new 
work  coming  to  his  installation. 
Furthermore,  the  installations  which 
are  just  starting  performance  programs 
would  have  a  baseline  for  setting 
initial  performance  goals. 


4.4  Lack  of  Statistical  CPE 
Techni  ques 

It  would  be  of  benefit  to  the  ADP 
manager  if  he  had  a  means  of  deter- 
mining what  effects  various  components 
of  the  system  and  workload  have  on  the 
service  that  his  installation  is 
providing.     How  does  one  bridge  the 
gap  between  measures  of  effectiveness 
in  meeting  service  goals  and  effi- 
ciency measures  in  resource  manage- 
ment?    How  does  the  performance  of 
certain  hardware  components  affect 
service  levels?     The  ability  to  cor- 
relate the  effects  of  computer  config- 
uration, operating  system  parameters, 
etc.     with  the  attainment  of  user 
service  requirements  has  not  been 
fully  studied  because  of  the  complex- 
ity of  the  interrelationships  and  also 
because  of  the  general  shortage  of 
data  which  would  help  in  the  eval- 
uation.    Instead  of  bridging  the  gap, 
the  ADP  manager  often  picks  one  goal 
by  deciding  what  is  more  important  to 
him  —  for  the  computer  to  run 
efficiently  or  for  the  computer  to 


serve  its  users  by  meeting  their 
goals.     Since  efficiency  is  often 
easier  to  attain  than  effectiveness, 
the  emphasis  has  been  historically  on 
efficiency.     However,  efficiency  of 
the  system  may  not  produce  the  service 
desired  by  the  users.     In  recent 
years,  more  emphasis  has  been  placed 
on  user  service  level  definition  and 
attainment.     But,  little  work  has  been 
done  in  bridging  the  gap  between  the 
two  alternatives  and  little  work  has 
been  done  on  statistical  techniques 
for  either  area. 

There  are  many  reasons  for 
specific  CPE  studies  but  they  all  have 
a  common  goals     to  improve  the  per- 
formance of  the  installation.  This 
goal  may  be  accomplished  by  making  the 
system  more  responsive  to  the  user 
requirements,  or  by  reducing  the  cost 
while  maintaining  the  same  service 
levels,  or  both.     The  problems  studied 
may  be  simple  with  simple  solutions  or 
may  be  extremely  complex  involving 
workload  characteristics  and  their 
effects  on  a  system  which  has  many 
interdependenc ies  and  interrela- 
tionships.    Simple  problems  may  be 
attacked  by  simple  graphical  methods 
while  others  may  require  more  rigorous 
and  precise  techniques.     The  latter 
studies,  however,  are  often  done  with- 
out the  rigor  and  preciseness  which 
comes  with  statistically  based 
techniques.     Instead  the  "eyeball" 
approach  is  used  v/here  practitioners 
subjectively  decide  whether  differ- 
ences in  a  measure  are  significant 
when  a  change  was  made  or  whether  a 
change  actually  had  the  desired 
effect.     Furthermore,  with  a  non- 
statistical  approach  nothing  can  be 
said  about  how  much  confidence  may  be 
placed   in  the  conclusions. 

There  are  two  basic  approaches  to 
analyzing  CPE  data:     graphical  and 
mathematical.     Graphical  techniques 
are  quite  straightforward  and  provide 
a  visual  means  of  detecting  patterns 
and  trends.     Graphical  techniques 
include  scatter  plots  for  detecting 
possible  relationships,  plots  of  time 
versus  service  and  resource  measure- 
ment on  the  same  graph  for  correla- 
tions or  maximums,  and  plots  for  trend 
analysis.     Graphs  will  often  indicate 
where  experimental  studies  should  be 
performed  by  visually  pointing  to 
possible  causal  relationships  between 
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system  and  service  variables  over 
time.     There  are  many  mathematical 
techniques  which  are  applied  to  CPE: 
correlation  analysis,  cluster  analy- 
sis, multiple  regression,  analytical 
models,  operational  analysis,  curve 
fitting,  time  series  analysis,  and 
response  surfaces.     Currently  there 
are  several  problems  with  using  any  of 
these  techniques. 

The  major  problem  is  that  these 
techniques  are  usually  based  on  under- 
lying assumptions  and  the  exact  nature 
—  statistical  or  otherwise  --  of 
computer  systems  and  workloads  is  not 
well  understood.     For  example,  regres- 
sion analysis  assumes  a  linear  rela- 
tionship between  the  dependent  vari- 
able and  the   independent  variables, 
equal  variances,  and  an  assumption 
that  the  independent  variables  are 
additive.     These  properties,  however, 
have  not  been  totally  verified   in  the 
general  computer  system  environment. 
A  second  problem  is  that  future  per- 
formance is  often  of  interest.  Model- 
ling techniques  are  the  most  often 
used  methods  for  predicting  perfor- 
mance in  growth  situations  where  the 
predicted  values  may  exceed  the 
current  operating  levels.  But 
analytical  modelling  has  very 
stringent  assumptions  and   is  still 
fairly  limited  in  modelling  the  true 
complexity  of  modern  computer  systems. 
The  most  stringent  assumptions  for 
analytical  modelling  are  removed  by 
operational  analysis  techniques  but 
many  of  the  queueing  parameters  needed 
are  currently  unavailable  from  most 
operating  systems.     The  techniques 
•discussed  above  are  not  exhaustive  but 
are  described  to   illustrate  the 
difficulties  and  problems  associated 
with  the  application  of  statistical 
analysis  to  CPE. 

With  all  of  the  problems  associ- 
ated with  good  CPE  studies,   it  is  no 
wonder   that  so  little  work  is  being 
done  at  installations.     The  more 
complex  techniques  are  statistically 
based  and  often  not  understood  or 
known  by  the  typical  analyst.  Little 
study  has  been  done  to  assess  the 
nature  of  the  computer  environment 
V i s-a-vi s  the  statistical  techniques 
being  used  today.  Furthermore, 
"cookbooks"  of  procedures  have  not 
been  developed  for  the  non-statis- 
tician analyst. 
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5.0  Proposed  Remedial  Work 

The  previous  section  discussed 
four  areas  which  were  seen  as  inhib- 
iting more  wide-spread  use  of  instal- 
lation performance  programs.  The 
inadequacies  increase  the  effort 
needed  to  develop  and  operate  computer 
performance  management  programs  at 
individual  ADP  installations.  If 
these  inadequacies  could  be  somewhat 
relieved,  less  installation  personnel 
and  fiscal  resources  would  be  required 
for  a  CPM  program.     Thus  the  use  of 
performance  programs  in  the  Federal 
sector  might  increase  and  the  benefits 
from  such  programs  could  be  realized. 
In  this  section,  we  will  discuss  some 
remedial  work  which  could  be  started 
to  alleviate  at  least  partially  the 
problems  mentioned  above:     lack  of 
measurabil ity ,   lack  of  well-defined 
terms,   lack  of  "good"  values,  and  lack 
of  statistical  CPE  techniques.  The 
proposed  remedies  involve  the  develop- 
ment of  standard  performance  measures, 
development  of  a  data  base  of  norm- 
ative standards  for  performance,  and 
development  of  statistical  techniques 
to  be  used  in  the  CPE  area. 


5.1  Standard  Performance  Measures 

Work  should  be  undertaken  to  de- 
velop a  list  of  standard  performance 
measures  and  definitions  for  these 
measures.     The  list  of  metrics  should 
satisfy  a  minimal  set  of  informational 
requirements  of  the  ADP  manager  in 
assessing  installation  performance  and 
should  be  vendor  independent. 

Since  the  establishment  of  a  set 
of  performance  metrics  does  not  ensure 
their  accessibility,  standard  report- 
ing formats  should  also  be  developed. 
Depending  upon  the  type  and  use  of  the 
data   item,   the  format  might  take  the 
form  of  a  record  on  a  file  or,  maybe, 
a  special  hardware  register  which 
could  be  monitored.     Most  measures 
which  are  needed  by  the  ADP  manager 
could  probably  be  written  to  a  special 
file  whose  format  would  be  standard 
across  vendor  lines.     However,  there 
are  some  types  of  data  which  are 
needed  but  which  would  cause  an  exces- 
sive drain  on  the  system  if  collected 
in  that  way.     For  these  measures,  an 
alternate  method  of  capturing  and 


recording  would  need  to  be  specified 
which  would  also  be  common  across 
vendor  lines. 

The  metrics  should  be  grouped 
according  to  purpose  and  use.  There 
are  two  reasons  for  considering 
multiple  groups.     First,  some  measures 
do  not  warrant  continuous  monitoring 
and  collection  but  are  needed  for 
specialized  studies.     The  overhead  in 
collecting  some  measurements  is  higher 
than  for  others  and  the  installation 
may  not  be  able  to  tolerate  the  drain 
on  resources.     Second,  depending  on 
the  sophistication  and  needs  of  a 
particular  installation,  not  all 
measures  would  be  of  interest  to  all 
installations.     Thus,  the  use  of 
multiple  groups  would  allow  the 
installation  to  tailor  the  collection 
of  performance  data  to  its  own  needs. 


5.2  Data  Base  for  Normative 
Performance 

Work  should  be  instituted  to 
determine  the  feasibility  of  develop- 
ing a  Government-wide  data  base  which 
would  contain  measures  of  installation 
performance  classified  by  type  of 
installation   (configuration)   and  type 
of  workload.     The  measures  would  have 
to  be  quantitative  if  automatic 
comparison  of  performance  is  to  be 
made  across  installation  types  and 
workloads.     The  problems  associated 
with  quantitatively  describing  work- 
loads,  installation  configurations, 
and  organizational  objectives  have  not 
yet  been  solved.     Without  such  quant- 
itative descriptions,  the  utility  of 
the  data  base  would  be  somewhat 
limited.     Some  types  of  measures  like 
hardware  reliability,   for  example,  are 
readily  quantifiable  and  normative 
values  would  be  useful.     By  examining 
averages  and  ranges  using  the  perfor- 
mance data  in  the  data  base,  optimum 
values  or  ranges  could  be  established 
which  would  provide  the  basis  for 
setting  individual  ADP  installation 
performance  goals.     These  values  would 
be  based  on  a  large  number  of  instal- 
lations and  would  therefore  provide  an 
additional  means  for  the  ADP  manager 
to  assess  his  installation's  perfor- 
mance . 
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Of  course,   if  such  a  system  is  to 
be  workable  it  must  help  the  ADP 
installations  and  require  little  in 
return.     It  must  have  a  well-designed 
collection  mechanism  which  would  not 
place  a  large  burden  on  the  individual 
installations.     The  system  must  also 
ensure  the  proper  feedback  of  inform- 
ation to  the  ADP  installations.  Last, 
if  the  data  is  to  be  useful,   it  must 
be  accurate.     It  is  anticipated  that 
installations  which  perceive  such  a 
data  base  as  a  threat  will  not  provide 
accurate  data  and  the  establishment  of 
normative  data  values  will  fail. 
Hence,  the  system  must  be  designed  to 
eliminate,  or  at  least  reduce,  the 
possibility  that  the  information 
supplied  by  the  individual  instal- 
lations might  be  used  against  them. 


5.3  Development  of  Statistical 
CPE  Techniques 

Work  should  be  started  to  develop 
and  validate  techniques  for  bridging 
the  gap  between  measures  of  effect- 
iveness and  resource  consumption 
measures.     The  environment  of  the 
computer   installation  and  its  inter- 
relationships need  to  he  studied  to 
determine  which  techniques  used  in 
other  control  environments   (such  as 
production  lines)    are  applicable  to 
the  computer  performance  area.  When 
such ' techn i ques  have  been  validated 
for  use  in  the  computer  environment, 
guides  need  to  be  prepared  for  the 
non-statistician  analyst  in  terms  that 
he  can  relate  to  and  understand. 

Work  should  be  continued  to 
develop  techniques  to  model  computer 
systems,  workloads,  and  service 
levels.     Such  work  is  needed  to  help 
the  ADP  manager   in  forecasting  his 
future  work  and  the  ability  of  his 
installation  to  handle  increased  or 
different  workloads.     Techniques  and 
tools  are  needed  which  are  relatively 
easy  to  use  and  which  use  data  which 
is  available  to  the  installation. 


because  they  lack  the  expertise  in 
their  staff  or  the  staff  is  not  large 
enough  to  spend  the  required  time  and 
because  there  can  be  rather  substan- 
tial costs  involved.     The  benefits 
from  the  three  proposals  above  come 
about  largely  because  of  their  poten- 
tial  for  reducing  the  effort  to 
implement  installation  performance 
programs.     The  standardization  of 
performance  measures  and  their 
reporting  format  would  allow  greater 
communication  between  installation 
practitioners  and  could  result  in 
better  solutions  to  common  problems 
being  developed  in  shorter  time 
periods.     The  approaches  to  solutions 
would  be  more  widely  known  and 
available.     Further,  the  standard- 
ization of  the  reporting  format  would 
allow  greater  portability  of  packages 
and  programs  which  are  currently 
rather  expensive  to  develop  both  in 
terms  of  time  and  money.     The  benefits 
of  performance  programs  vary  from 
installation  to  installation,  but  no 
benefits  can  be  realized   if  no  such 
activity  is  being  done.     By  reducing 
the  effort  to  start  and  participate  in 
performance  programs  at  installations, 
it  is  hoped   that  more  ADP  managers 
will  be  willing  to   institute  such 
programs  at  their  sites. 


5.4  Summmary 

We  believe  that  many  Federal  ADP 
installations  perform  little  or  no 
work  in  installation  performance 
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Appendix  A  -  Informational  Model 


Based  on  the  functions  discussed  in  Section  2,  the  following 
informational  model  is  presented.  There  is  no  attempt  to  make  this 
model  complete,  or  uniformly  detailed,  but  the  intent  is  to 
illustrate  the  magnitude  and  variety  of  information  used  in 
supporting  the  installation  performance  program.  Usually  the 
detail  levels  are  not  given  since  the  exact  detail  information 
would  depend  on  the  objective  of  the  function;  for  example,  the 
type  of  optimization  in  workload  management  -  program,  file 
handling,  etc.     -  determines  the  exact  measures  to  be  made. 


I.  Upper  Management 

A.  Planning 

1.  Configuration  alternatives 

2.  Pricing  strategy 

*  workload  forecast 

*  budget  projection 

*  new  service  levels 

3.  Organization  objectives 

*  type  of  processing 

*  priority 

*  schedule 

B,  Operations 

1.  Fiscal  Accounting 

*  revenue 

*  project 

*  resources 

*  budget 

*  expenditures 

*  salaries 

*  supplies 

*  equipment 

*  packages 

*  maintenance 

*  space/utilities 

*  pricing  strategy 

*  recovery 

*  behavior  modification 

2.  Resources 

*  total  system  capacity 

*  personnel 

3.  Workload 

*  organizational  unit 

*  trends 

4.  Service  levels 

*  timeliness 

*  reliability 

*  availability 

*  cost  when  level  not  met 
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1 


User  Community 


A.  User  Satisfaction 

1.  Survey 

2.  Suggestion  forms 

B.  Setting  Service.  Levels  and  Pricing  Strategy 

1.  Workload  forecast   (resource  units) 

2.  Revenue  projection 

C.  Workload  Management 

1.  Audit   (delete  or  reschedule) 

*  project(s)  supported 

*  dependencies 

*  age  of  application 

*  age  of  version 

*  run  frequency 

*  time  of  day 

*  resources  requested 

*  resources  consumed 

2.  Optimize 

*  execution  characteristics 

3.  Quality  assurance 

D.  User  Proficiency 

1.  Trouble  desk 

*  rel iabil ity 

*  availability  (system) 

*  availability  (personnel) 

*  types  of  problems 

*  utilities  needed 

2.  User  groups 

*  service  level  tracking 

*  workload  statistics 

3.  Technical  support 

*  modules  being  developed 

*  use 

*  schedule 

*  revenue 

*  staff  time  expended 

*  user 

*  utility 

*  system 

4.  Communication 

*  service  level  tracking 

*  cost   (chargeback)  information 

*  new  packages,  upgrades,  etc. 

*  system  schedule  changes 

*  training  notices 

*  efficiencies  suggestions 

5.  Training 

*  workload  statistics 

*  reliability  (user  errors) 

*  rates 

*  type 

*  jobs  affected 

*  new  procedures,  packages 
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E.  Planning 

1.  Service  requirements 

2.  Conversion  impact  areas 

*  allocatable  devices 

*  magnetic  tapes  • 

*  disk  packs 

*  data  files 

*  data  base  management  systems 

*  internal  formatted 

*  programs 

*  non-standard  features 

*  non-standard  languages 

3.  Workload  forecast 

*  functional 

*  resource 


III,     Resource  Management 

A.  Operational  and  Procedural 

1.  Dispatching  functions 

*  staff  time 

*  volumes 

*  cards  handled 

*  pages  handled 

*  poor  quality  output 

2.  Schedules 

*  availability  of  staff  and  system 

*  schedule  versus  actual 

3.  Errors   (system  and  personnel) 

*  number  by  type 

*  time  to  fix  or  restore 

*  fatal  to  system  or  subsystem 

*  which  jobs  affected 

*  direct 

*  indirect 

*  cost  to  projects 

*  wasted  processing  time 

*  wasted  staff  time 

*  cost  to  installation 

*  effect  of  user  practices 

*  file  save  rate 

*  backup/restore  activity 

4.  Backup/restore 

*  schedule 

*  frequency 

*  resources  required 

5.  Training  (operators) 

*  errors 
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Computer  System 

1.  Control  limits 

*  utilization 

*  total  system 

*  device 

*  time  periods 

*  subsystems 

*  availability 

*  time 

*  down 

*  scheduled 

*  unscheduled 

*  diagnostic 

*  waiting  for  repair 

*  repair 

*  up 

*  completely  available 

*  subsystem  down 

*  unstaffed 

*  total  system 

*  hardware  groups 

*  software  modules 

*  responsiveness 

*  turnaround  time 

*  time-sharing  response 

*  number  of  jobs  ontime/late 

*  throughput 

*  file 

*  space 

*  distribution 

*  allocatable  device 

*  number  busy 

*  time  busy 

*  contention  and  overlaps 

2.  Tuning 

*  utilization 

*  by  component 

*  by  device 

*  productivity 

*  subsystem  activity 

*  communication  activity 

*  timesharing 

*  remote  batch 

*  transaction 

*  line  speeds 

*  profiles 

*  port  activity 

*  service 

*  system  queues 

*  user  requested 

*  input 

*  output 

*  allocatable  device 

*  processor 

*  misc.  internal 

*  file 

*  organization 

*  placement 
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(2.  Tuning  continued) 

*  allocatable  device 

*  service  -  utilization  relationship 

*  system  interrelations 

*  contention 

*  overlaps 

*  concurrency 
3.  Forecasting 

*  workload  characterization 

*  function 

*  batch  (over-the-counter) 

*  arrival  distribution 

*  priority 

*  dependency 

*  remote  batch  (job  entry) 

*  origination 

*  priority 

*  dependency 

*  timesharing 

*  number  sessions 

*  functions 

*  order  of  functions 

*  volume  of  characters 

*  transaction 

*  number  and  type 

*  volume  of  characters 

*  language 

*  file  storage 

*  permanent  or  scratch 

*  resources  required 

*  workload  -  service  relationship 

*  workload  -  utilization  relationship 

*  service  -  utilization  relationship 

*  system  interrelations 

*  queue 

*  contention 

*  overlaps 
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Vendor 

A.  Hardware  Reliability 

1.  Error 

*  jobs  affected 

*  which 

*  cost  to  project 

*  cost  to  installation 

*  rates 

*  intervals 

2.  Retry 

*  rates 

*  intervals 

3.  Maintenance 

*  schedule 

*  unscheduled 

*  response 

*  delays 

*  error  profile 

B.  Software  Reliability 

1.  Error 

*  jobs  affected 

*  which 

*  cost  to  project 

*  cost  to  installation 

*  rates 

*  intervals 

*  time  to  fix 

*  release/version 

*  between  errors 

2.  Retry 

*  rates 

*  intervals 

3.  Maintenance 

*  time  between  releases 

*  event  profile 

*  regression 
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